feature: add discover-azure-skills skill - #3053
feature: add discover-azure-skills skill#3053JasonYeMSFT (JasonYeMSFT) wants to merge 8 commits into
Conversation
Example UX in Copilot CLI. |
There was a problem hiding this comment.
Pull request overview
Adds a new discover-azure-skills agent skill intended to help users discover relevant Azure-related skills/plugins and includes test registration plus a routing eval to validate invocation behavior.
Changes:
- Registers
discover-azure-skillsintests/skills.jsonand adds it to the integration schedule. - Adds a new skill package under
plugins/azure-skills/skills/discover-azure-skillswith install reference docs. - Adds a new Vally integration eval suite for routing/invocation coverage.
Show a summary per file
| File | Description |
|---|---|
tests/skills.json |
Adds the new skill to the azure plugin skill list and integration schedule. |
plugins/azure-skills/skills/discover-azure-skills/version.json |
Introduces per-skill NBGV version configuration. |
plugins/azure-skills/skills/discover-azure-skills/SKILL.md |
Defines the discovery workflow and links to install instructions. |
plugins/azure-skills/skills/discover-azure-skills/references/install/other.md |
Provides generic install instructions via npx skills add. |
plugins/azure-skills/skills/discover-azure-skills/references/install/copilot-cli.md |
Provides Copilot CLI installation steps. |
plugins/azure-skills/skills/discover-azure-skills/references/install/claude-code.md |
Provides Claude Code installation steps. |
evals/azure-skills/discover-azure-skills/eval.yaml |
Adds an integration routing eval validating skill invocation across prompts. |
Review details
Tip
Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 7/7 changed files
- Comments generated: 8
- Review effort level: Lite
| 1. Install each skill using `skills` package | ||
|
|
||
| ``` | ||
| npx skills add https://github.com/microsoft/azure-skills/tree/main/.github/plugins/{plugin-dirname}/skills/{skill-name} |
There was a problem hiding this comment.
Notes for reviewers:
- How do you feel about requiring npx? I haven't found any other well-known tool that handles skill installation as well as skills.sh while being more multi-platform friendly.
- VS Code is currently categorized under "other" because its built-in mechanism doesn't support specifying which plugin to install from a repo. Allow users to specify which plugin to install in "Install Plugin from Source" command vscode#330047
- Skills installed in this way won't send telemetry since hooks won't be installed.
|
Does this skill require me to ask for discovery intentionally? Would the customer need to do "discover" explicitly? Should the skill be more of a guide that happens based on intent to do something Azure? |
I expect the user to at least express the intent to learn how to do something in Azure to trigger this skill. I intend to avoid having the skill triggered when it's not the case. We can discuss more details of the design in this issue. https://github.com/microsoft/GitHub-Copilot-for-Azure-pr/issues/358 |
c8fcb25 to
0e1c899
Compare
0d5b53a to
4c7aa2f
Compare
|
I was able to implement the discover-azure-skills skill to make it work like a catch-all skill when the user issue commands for Azure tasks without explicitly asking for finding new skills. However, given that we only have one additional plugin available right now, the overhead we introduce by making agents trying hard to find new skills is very significant. I still prefer taking an conservative approach for now, grow the platform to have more plugins/skills and tune the skill to make more aggressive recommendations later. |
| Write-Output '{"hookSpecificOutput": {"hookEventName": "SessionStart", "additionalContext": "Make sure to use discover-azure-skills once when the user asks you to execute a task related to Azure if there is no relevant skill/mcp tool in the context. Avoid using discover-azure-skills when the user asks informational questions on how to do something."}}' | ||
|
|
||
| # Output success to stdout (required by hooks) | ||
| Write-Success No newline at end of file |
There was a problem hiding this comment.
Write-Success is defined only inside track-telemetry.ps1 and this script does not import it. Every Windows Copilot CLI or VS Code session therefore emits a command-not-found error during SessionStart after writing the context. Replace this with a local success response or an explicit exit 0 so the hook completes cleanly.
| @@ -0,0 +1,3 @@ | |||
| #!/bin/bash | |||
There was a problem hiding this comment.
This script has a shebang but is committed as mode 100644, while the existing hook script is executable. The Copilot CLI, Claude Code, and Cursor manifests invoke it as a command, so Unix sessions can fail with Permission denied before the discovery context is emitted. Mark this file executable in Git.
Description
A new discover-azure-skills skill to help discover available skills among all the plugins we own. The discover-azure-skills skill can be triggered when the user explicitly asks the agent to find a new skill/plugin or asks the agent to do an Azure related task. However, it won't be triggered when the user asks informational questions on how to do an Azure related task.
To achieve this trigger condition, a SessionStart hook that injects a higher level instruction is necessary. This PR adds a new SessionStart event hook that injects instructions as additional context to the session. The injected instructions costs ~64 tokens.
The plugin build script is refactored to support building per-plugin hooks. The repo now maintains a shared set of hooks and optionally hooks per plugin. The build script can merge the shared hooks and the per-plugin hooks at compile time. Right now, we share telemetry hook for all plugins and only add this new hook for discover skills to the azure-skills plugin.
Checklist
cd tests && npm test)fix:,feat:,feature:,chore:,misc:,test:,eval:tests/,npm run test:integration -- <skill>ornpm run test:vally -- --skill <skill>)Related Issues
resolves #2794